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ABSTRACT: The current study explored relations between non-cognitive traits (Grit, Learning 
Orientation, Performance Orientation), reading skill, and performance across three 
experiments conducted in the context of two intelligent tutoring systems, iSTART and 
Writing Pal. Results showed that learning outcomes (comprehension score, holistic essay 
score) were moderately to strongly correlated with reading skill. In contrast, these outcomes 
only weakly correlated with learning orientation (LO) and were unrelated to either Grit or 
performance orientation (PO). Further, regression analyses indicated that none of the non- 
cognitive traits predicted learning gains. We open the discussion of these findings in the 
context of theoretical perspectives, construct validity and reliability, and large scale 
assessment. 
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1 INTRODUCTION 


In addition to the contributions of cognitive skills, non-cognitive factors (e.g., motivation and affect) 
have been found to predict academic success across a variety of domains and age levels (Pintrich, 
2003). While researchers have focused on a range of individual differences related to self-regulation 
(e.g., metacognition or monitoring, motivation), one of most widely talked about non-cognitive 
factors in the mainstream and in education is Grit. Initial research indicated that this higher-order 
trait was prevalent in successful people from a variety of domains (e.g., US military, National Spelling 
Bee) and that Grit predicted success above and beyond “talent” or cognitive ability (Duckworth et 
al., 2011). In addition to Grit, researchers have also been interested in the effects of learning 
orientation (LO) and performance orientation (PO). LO is the degree to which a student is 
intrinsically motivated to learn new things. PO reflects extrinsic motivation related to tangible gains. 
LO has been found to be a stronger predictor of long-term educational success than PO (Fortune et 
al., 2005, Rosé et al., 2004). However, PO has stronger relations to performance on short-term tasks 
(Jha & Bhattachayya, 2013). Thus, both may be relevant for different aspects of learning. 


Due to the mainstream attention on Grit, there has been a boom in developing interventions to 
increase non-cognitive skills and expanding methods of evaluating them. More recently, however, 
Grit has come under scrutiny, with a meta-analysis indicating a lack of construct validity and stability, 
as well as little effect of grit interventions on performance and success (Credé et al., 2017). Thus, 
discussions of the effects of non-cognitive constructs at scale should be preceded by additional 
evaluations of their relations to learning in the context of on-line environments. Indeed, there is a 
growing body of conflicting evidence for the effect of these non-cognitive traits in terms of online 
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course completion (Phillips-Martinez, 2017; Wang, 2017) and behaviors within online educational 
environments (e.g., Baker et al., 2008). The current study adds to this literature by investigating how 
non-cognitive factors 1) relate to cognitive factors that are relevant to performance and 2) affect 
learning outcomes. 


The current study examines the effects of these non-cognitive factors in the context of two 
intelligent tutoring systems (ITSs): Interactive Strategy Training for Active Reading and Thinking 
(iSTART; McNamara et al., 2004) and the Writing Pal (W-Pal; Roscoe & McNamara, 2013). These ITSs 
reflect a growing cadre of educational technologies that support students’ reading and writing skills 
(Passonneau et al., 2018). In both systems, students watch instructional videos and then engage in 
game-based practice. Natural language processing (NLP) is used to provide both summative and 
formative feedback. iSTART offers reading comprehension strategy training and has been shown to 
increase students’ understanding of challenging science texts across middle, high school, and college 
populations (e.g., Jackson & McNamara, 2013; Magliano et al., 2005). W-Pal combines intelligent 
tutoring and automated writing evaluation (AWE) to enhance adolescents’ writing quality and 
knowledge of writing strategies (Roscoe et al., 2013). Importantly, mastery of these literacy skills 
depends on repeated, deliberate practice over extended time periods. According to motivation 
theories, students with higher Grit and other motivation-related traits should be more engaged, 
persistent, and deliberate with their practice in iSTART and W-Pal which would, in turn, enhance the 
efficacy of the training. The purpose of this study was to explore this hypothesis. 


To summarize, this study had two objectives. Our first objective was to investigate the relations 
between these non-cognitive traits (Grit, LO, PO) and a cognitive skill shown to predict 
comprehension and writing success (i.e., reading skill). The second objective was to explore how 
these traits influence learning gains after using iSTART and W-Pal. 


2 DATA SOURCES 


The current study analyzes three extant data sets in which high school students (total n = 368) 
demonstrated pretest to posttest learning gains as well as completed: 1) assessments of non- 
cognitive and cognitive individual differences, 2) pretest and posttest learning assessments, and 3) 
multiple sessions of iSTART or W-Pal training. The individual difference measures were administered 
at pretest and included the 8-item short Grit scale (GRIT-S; Duckworth & Quinn, 2009), the 11-item 
Learning Orientation (LO) and Performance Orientation (PO) Scale (Jna & Bhattachayya, 2013), and 
the Gates-MacGinitie Reading Test (GMRT; MacGinitie & MacGinitie, 1989) as a standardized 
measure of reading comprehension skill. The learning outcome in iSTART A and B was a 
comprehension test score. At pretest and posttest, participants read a scientific passage and then 
answered open-ended text-based and inference comprehension items. Trained researchers scored 
each response for accuracy (0-1; Cohen’s Kappas > .80). In the W-Pal study, the learning outcome 
was a holistic essay score. Participants were given 25 minutes to write a persuasive essay at pretest 
and posttest. Four expert raters (English graduate students) trained in the rating procedure provided 
a holistic score (0-6; exact agreements > .45, adjacent agreements > .97, weighted Kappas > .68). 


In iSTART A, participants (n = 149) engaged in 4 hours of training across two sessions and practiced 
in either a game or non-game environment (Jacovina et al., 2015). In iSTART B, participants (n = 116) 
engaged in 6 hours of training across three sessions and received different types of metacognitive 
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prompting (McCarthy et al., 2017). Though both experiments demonstrated overall pretest to 
posttest gains as a function of training, there were no effects of the experimental manipulations. 
Thus, the analyses for this study are collapsed across these conditions. In the W-Pal study, 
participants (n = 103) received approximately 6 hours of training across four sessions. 


3 RESULTS 


Correlational analyses on the individual difference measures indicated that LO and PO were 
correlated. However, only LO significantly correlated with GMRT (Table 1). Notably, across the three 
studies, the relations between Grit and the other measures were weak and inconsistent. 


Table 1: Correlations in Three Studies between Individual Difference Measures [Learning 
Orientation (LO), Performance Orientation (PO), Grit] and Gates MacGinitie Reading Test (GMRT). 


iSTART A iSTART B W-Pal 
1 2 3 1 2 3 1 2 3 
1.LO 1 1 1 
2.PO aS 1 32° 1 30° 1 
3. Grit 16 -.14 1 19° -.32° 1 13 .00 1 
4.GMRT 41 03 09 a5 16 202 A6 26° =18 


Table 2 shows correlations between the individual differences and the learning outcomes at pretest 
and posttest (comprehension test score, essay score). GMRT was strongly correlated with 
comprehension scores at both pretest and posttest. Weak correlations were observed between LO 
and the learning outcomes. Neither PO nor Grit were correlated with these outcomes. 


Table 2: Correlations in Three Studies between Individual Differences 
and Pretest and Posttest Outcomes. 


iSTART A iSTART B W-Pal 
(Comprehension) (Comprehension) (Essay) 
Pretest Posttest Pretest Posttest Transfer Pretest Posttest 
LO 35° Al 24° 20° 24° 33° 32° 
PO 10 03 12 -.02 13 .20 ll 
Grit .03 18° -.05 -.01 -.16 -.08 .03 
GMRT 66 70° 53° ST 58 65" AS” 


Regression analyses examined how the motivational factors predicted learning gains. To account for 
the fact that participants who score lower at pretest have more room for improvement, we 
calculated relative gain scores [e.g., (proportion correct at posttest — proportion correct at pretest)/(1 
— proportion correct at pretest)]. Regression models with LO, PO, and Grit entered simultaneously 
indicated that none of these measures accounted for significant variance in learning gains (iSTART A: 
R? = .02, F(3, 140) = 1.08, ns; iSTART B: R? = .01, F< 1.00; W-Pal: R? = .02, F(3, 96) = 1.35, ns). 


4 DISCUSSION 


This study examined the effects of non-cognitive traits on performance and learning gains in two 
ITSs that support literacy: iSTART and W-Pal. It was found that Grit and PO were not significantly 
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correlated with reading skill, comprehension score, or essay score. This study is consistent with 
extant research indicating null effects of performance orientation on in-system choices (e.g., Baker 
et al., 2008) and extends the work by demonstrating null effects on post-training literacy outcomes. 
While LO was correlated with comprehension test performance and essay score, it did not 
significantly predict learning gains. These results indicate that these higher-order non-cognitive 
factors are not driving forces of reading skill or success in these reading and writing ITSs. 


There are several potential explanations for these results. One is that non-cognitive (motivational) 
factors are simply unrelated to long-term learning gains in these environments. Indeed, this is 
consistent with extant work in both iSTART and W-Pal showing that game-based practice increases 
motivation and enjoyment, but does not directly impact learning (Jackson & McNamara, 2013; 
Proske et al., 2014). If this is the case, then it behooves researchers and educators to further 
understand relations between cognitive and non-cognitive skills in educational environments, and to 
strive to report non-significant relations as well as significant ones. An alternative explanation is that 
the traits themselves do predict learning, but that the short surveys used to assess the underlying 
constructs are unreliable. If this is the case, analytics and data mining approaches may provide 
stealth assessment of these traits that is more valid than the current self-report measures (e.g., 
Fancsali et al., 2014). A final possibility is that non-cognitive factors may be important, but more so 
for moment-to-moment actions and decisions than overarching success. As such, it may remain of 
value to evaluate situational interest, motivation, or affect during interactions in these environments 
(see D’Mello, 2013 for review). We intend to explore and further discuss what other kinds of non- 
cognitive factors might be more predictive of learning gains in this type of on-line environment and 
where these effects might be most evident. In sum, we welcome continued research into 
assessment of and interventions for non-cognitive factors. However, based on our findings, we 
caution researchers, designers, and educators to first consider the impacts of these factors as a 
means of prioritizing the central goal of enhancing learning. 
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